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SUMMARY 


Problem 


The  increasing  complexity  and  sophistication  of  modern  weapon  and  support  systems 
are  resulting  in  massive  volumes  of  technical  operation  and  maintenance  information. 
Electronic  information  delivery  holds  potential  for  rapid  access  to  needed  information 
stored  on  compact  memory  units.  There  have  been  only  limited  comparisons,  however, 
between  the  effectiveness  and  efficiency  of  electronic  information  delivery  and  conven¬ 
tional  paper-based  technical  manuals. 

Objective 

The  objective  of  this  study  was  to  extend  the  results  of  an  earlier  Air  Force  study  and 
compare  the  troubleshooting  performance  of  military  technicians  who  obtained 
information  from  conventional,  paper-based  maintenance  manuals  and  from  electronic 
devices. 

Procedure 

Four  troubleshooting  tasks  were  administered  to  36  technicians,  12  each  from  the  Air 
Force,  Navy,  and  Marine  Corps.  The  tasks  consisted  of  identifying  a  fault  introduced  into 
a  circuit  relay  or  discrete  component  on  a  printed  circuit  card  used  in  the  RT-728A/APX- 
64(V)  radio  receiver  transmitter.  Two  tasks  were  performed  using  conventional  manuals 
and  two  using  information  presented  on  a  GRiD  Compass  II  computer.  Presentation 
combinations  were  balanced  to  avoid  experimental  bias.  Independent  variables  of 
technician  experience  level  and  information  delivery  method  were  arranged  in  a  2  x  2, 
between-subjects  multiple  analysis  of  variance  (MANOVA)  design.  Dependent  variables 
included  measures  of  time  and  efficiency.  User  evaluations  were  obtained  by  means  of 
questionnaires  and  structured  interviews. 

Results 


Significant  differences  were  found  with  regard  to  time  to  identify  faulted  card,  total 
problem  solution  time,  and  number  of  false  item  replacements,  all  of  which  were  less  for 
electronic  than  for  conventional  information  delivery.  Significant  differences  were  also 
found  with  regard  to  number  of  test  points  checked,  total  number  of  tests  performed,  and 
number  of  problems  solved,  all  of  which  were  greater  for  electronic  than  for  conventional 
information  delivery.  Thus,  with  electronic  information  delivery,  technicians  performed 
more  thorough  and  successful  troubleshooting  in  less  time.  Technician  evaluations  of 
electronic  information  delivery  were  also  consistently  favorable.  The  only  major 
criticism  of  the  computer  concerned  the  inadequacy  of  screen  size  for  displaying 
schematics. 

Conclusion 


Electronic  information  delivery  can  be  an  effective  and  efficient  means  of  presenting 
technical  maintenance  information.  To  date,  however,  most  research  has  involved  only 
limited  data  bases.  Future  research  needs  to  address  the  effects  of  large  technical  data 
bases  and  the  various  ways  of  accessing  the  different  types  of  information  in  those  data 
bases. 
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INTRODUCTION 


Problem 


With  the  growing  complexity  of  military  systems,  there  is  a  concomitant  increase  in 
the  volume  of  technical  information  needed  for  their  operation  and  maintenance.  For 
example,  today's  F/A-18  Hornet  aircraft  requires  approximately  500,000  pages  of  techni¬ 
cal  documentation  compared  to  1,800  pages  for  a  jet  fighter  aircraft  introduced  in  1950. 

One  method  for  coping  with  increased  equipment  complexity  and  the  amount  of 
technical  information  supporting  it  has  been  the  assignment  of  technicians  with  different 
types  of  training  and  experience  to  individual  subsystems  within  a  major  system  such  as  an 
aircraft.  Unfortunately,  since  the  population  from  which  available  technicians  are  drawn 
is  decreasing  (Binkin,  1986),  it  will  not  be  effective  to  have  technicians  specialize  in 
single  systems  or  subsystems  in  the  future.  Rather,  quick  access  to  accurate  technical 
information  by  all  technicians  will  be  a  necessity  for  the  maintenance  of  modern  weapon 
systems. 

Electronic  presentation  techniques  offer  an  approach  to  quick  access  of  technical 
information.  The  Department  of  Defense  recognizes  this  in  calling  for  the  "extensive 
study  of  user  interaction  with  non-paper  maintenance  aids"  and  "research,  experimenta¬ 
tion,  and  field  trial  experience  ...  to  find  out  what  form  of  displays  are  best  for  specific 
situations"  (Riddell,  Gunkel,  Beiser,  Goldstein,  it  Lepisto,  1985).  Of  critical  interest  is 
the  effectiveness  of  using  non-paper  maintenance  aids  and  displays  for  quick  access  to 
technical  information  for  fault  isolation.  While  the  Air  Force  has  demonstrated  the 
feasibility  of  electronic  delivery  of  technical  information  (Clay,  1986),  performance  data 
comparing  the  effectiveness  and  efficiency  of  electronically  presented  information  to 
those  of  the  conventional  paper  technical  manuals  have  been  limited. 

Objective 

The  objective  of  this  study  was  to  extend  the  results  of  an  earlier  Air  force  study  and 
compare  the  troubleshooting  performance  of  military  technicians  who  obtained 
information  from  conventional,  paper-based  maintenance  manuals  and  from  electronic 
devices. 

Background 

The  present  effort  is  part  of  a  larger  research  and  development  project  titled 
"Computer-based  Aids  for  Troubleshooting  (CBAT)."  The  goal  of  CBAT  is  to  design  and 
evaluate  electronic  presentation  techniques  to  ensure  that  required  troubleshooting 
information  is  readily  available  to,  and  accessible  by,  the  technician  during  corrective 
maintenance.  The  present  evaluation  was  an  extension  of  an  earlier  Air  Force  Human 
Resources  Laboratory  (AFHRL)  study  in  which  electronic  presentation  of  technical 
information  was  used  to  support  isolation  of  a  single  fault  (Clay,  1986). 

In  the  AFHRL  study,  a  prototype  computer-based  maintenance  aid  system  (CMAS) 
was  developed  to  evaluate  technical  data  presentation  and  user  interface  techniques.  An 
off-the-shelf  computer  was  used  as  the  prototype  delivery  device.  Software  was 
developed  to  store  and  present  technical  information  in  an  integrated  text/graphics 
format.  A  preliminary  evaluation  of  this  data  base  was  accomplished  by  having  eight  Air 
Force  technicians  perform  assigned  maintenance  tasks  using  technical  information  pre¬ 
sented  via  CMAS.  In  general,  the  technicians  expressed  positive  comments  about  the  use 
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of  CMAS  to  support  maintenance.  Since  the  primary  purpose  of  the  AFHRL  study  was  to 
demonstrate  the  concept  of  electronic  delivery  of  technical  information,  only  limited  data 
were  obtained  comparing  CMAS  to  conventional  technical  manuals  in  isolating  a  fault. 

While  several  studies  have  examined  effects  of  technician  experience  level  on  the  use 
of  different  methods  for  presenting  technical  information  (e.g.,  Elliott  &  3oyce,  1971; 
Serendipity,  Inc.,  1969),  the  impact  of  this  variable  when  using  electronic  presentation  as 
an  alternative  to  the  paper-based  technical  manual  has  not  been  sufficiently  researched. 
The  results  from  the  present  study  will  be  used  in  conjunction  with  those  from  the  AFHRL 
study  to  document  the  advantages  and  disadvantages  of  electronic  delivery  of  technical 
information  for  both  experienced  and  inexperienced  maintenance  personnel. 


APPROACH 

Test  Vehicle  and  Troubleshooting  Tasks 

The  RT-728A/APX-6<*(V)  radio  receiver-transmitter,  which  served  as  the  test  vehicle 
for  the  AFHRL  study,  was  also  used  as  the  test  vehicle  in  the  present  effort.  This 
equipment  provides  ground-based  control  centers  with  aircraft  location,  altitude,  and 
identification  (friend  or  foe)  data.  As  shown  in  Table  1,  the  RT-728A/APX-64(V)  is 
currently  installed  in  a  variety  of  Navy,  Air  Force,  and  Marine  Corps  aircraft. 


Table  1 

Use  of  the  RT-728A/APX-64(V)  by  Navy/Marine  Corps/ 
Air  Force  Aircraft 


Navy 

Marine  Corps 

CH-53  A&D 

CH-53  A&D 

A-4F 

OA-4M 

EA-4F 

OV-1DA 

A-6B  (Reserves) 

OV-1DD 

T-2B  &  C 

7A-7C 

Air  Force 

TA-4F 

TA-4J 

B-52 

KC-135 

C-137 

H-3 

CH-53 

Four  RT-728A/APX-6*»(V)  printed  circuit  card  failures  were  used  as  troubleshooting 
problems  in  the  present  effort.  Of  these,  one  was  identical  to  that  used  in  the  AFHRL 
study.  The  remainder  were  chosen  with  the  assistance  of  senior  RT-728A/APX-64(V) 
maintenance  instructors  at  the  Naval  Air  Maintenance  Training  Group  Detachment  1028 
in  Tustin,  California. 


The  printed  circuit  card  malfunctions  were  introduced  by  faulting  circuit  relays  or 
discrete  components.  Additionally,  precautions  were  taken  to  ensure  that  (1)  the  faulted 
components  could  not  be  easily  identified  on  the  basis  of  visual  inspection  alone,  (2)  the 
casualty  symptoms/failure  indications  for  any  given  problem  were  consistent  upon 
successive  administration  of  the  problem,  and  (3)  sufficient  information  was  available 
from  both  the  automated  system  and  the  current  RT -728A/APX-64(V)  technical  manual 
for  isolating  each  printed  circuit  card  failure  to  the  component  level. 

Technical  Information  Delivery  Systems 

The  two  delivery  methods  used  to  present  fault  detection  and  fault  isolation 
information  were  the  technical  manual  for  the  RT-728A/APX-64(V)  and  an  off-the-shelf 
computer  with  an  expanded  version  of  the  AFHRL  CMAS  technical  information  data  base. 

The  computer  was  a  GRiD  Compass  II,  model  1 139.1  The  GRiD  was  used  in  the 
earlier  AFHRL  study  because  of  its  small  size  and  powerful  capabilities.  Since  the 
operating  system  software  and  the  available  data  base  were  somewhat  uniquely  tailored  to 
the  GRiD,  the  newly  developed  material  was  added  to  the  existing  data  base. 

AFHRL  had  developed  the  software  to  store  and  present  technical  manual  data  on  the 
GRiD.  Functions  provided  by  the  software  included:  (1)  scrolling  to  portions  of  a  diagram 
too  large  to  be  displayed  on  the  screen,  (2)  direct  access  to  parts  of  the  data  base  without 
using  menus,  (3)  automatic  branching  to  a  specific  kind  of  information  after  a  decision  had 
been  made,  (4)  display  of  available  options,  and  (5)  availability  of  two  levels  of  detail  for 
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Disconnect  cable  fron  RN/UPM- 
137R  oscilloscope  CHRN  R  VIDEO 
IN  (1). 

Connect  XI  probe  fron  oscillo¬ 
scope  CHRN  R  VIDEO  IN  (1)  to 
test  point  R5R1TP4  (2). 

Connect  XI  probe  fron  oscillo¬ 
scope  CHRN  B  VIDEO  IN  (3)  to 
test  point  R5R1TP3  (4). 
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Figure  1.  An  example  of  the  integrated  text/graphics  format  of  the 
electronic  data  base.  The  MORE  level  of  detail  is 
presented  here. 


1  Identification  of  the  equipment  is  for  documentation  only  and  does  not  imply  endorse¬ 
ment. 


procedural  information.  The  general  structure  of  the  electronic  data  base  was  an 
integrated  text/graphics  format,  as  shown  in  Figure  1.  As  stated,  procedures  in  the 
electronic  data  base  were  developed  for  two  levels  of  detail.  The  MORE  level  contained 
detailed  text  descriptions  along  with  supporting  graphics.  The  LESS  level  was  an 
abbreviated  version  of  MORE,  a  checklist  with  less  detailed  graphics.  MORE  was 
designed  for  inexperienced  technicians,  while  LESS  was  intended  for  more  experienced 
personnel. 

Changes  to  the  original  CMAS  data  base  included  (1)  additional  schematic  diagrams, 
(2)  detailed,  illustrated  parts  breakdown  information,  and  (3)  troubleshooting  procedures 
necessary  to  cover  the  four  problems.  These  changes  were  developed  and  validated  with 
the  assistance  of  RT-728A/APX-64(V)  maintenance  technicians  at  the  Air  Intermediate 
Maintenance  Department,  Naval  Air  Station,  North  Island,  in  San  Diego,  California. 

Participants 

Thirty-six  RT-728A/APX-64(V)  technicians  for  the  Air  Force,  Marine  Corps,  and 
Navy  assigned  to  four  intermediate-level  avionics  maintenance  facilities  in  California 
served  as  participants.  Six  of  the  12  Marine  Corps  technicians  were  from  Headquarters 
and  Maintenance  Squadron-13,  Marine  Corps  Air  Station  in  El  Toro;  the  remainder  were 
from  Headquarters  and  Maintenance  Squadron-16,  Marine  Corps  Air  Station  in  Tustin. 
The  12  Air  Force  technicians  were  assigned  to  the  93rd  Avionics  Maintenance  Squadron, 
Castle  Air  Force  Base  in  Merced.  The  12  Navy  technicians  were  assigned  to  the  Air 
Intermediate  Maintenance  Department,  Naval  Air  Station,  Miramar,  in  San  Diego. 

Since  level  of  experience  has  been  shown  to  have  an  effect  on  performance 
(Serendipity,  Inc.,  1969),  the  participants  at  each  facility  were  classified  into  two  groups 
consisting  of  equal  numbers  of  experienced  and  inexperienced  technicians.  Group 
assignment  was  based  on  a  combination  of  relevant  field  experience  in  the  maintenance  of 
the  RT-728A/APX-64(V)  (i.e.,  1  or  more  years  for  the  experienced  group;  less  than  1  year 
for  the  inexperienced  group),  and  judgments  of  immediate  work  supervisors  concerning  the 
technician's  maintenance  qualifications. 

Participation  in  the  testing  was  voluntary.  No  one  declined  to  participate,  and  all 
participants  appeared  interested  and  cooperative  throughout  the  test  sessions. 

Procedure 

A  detailed  test  administrator's  guide  was  prepared  to  standardize  the  presentation  of 
instructions  and  troubleshooting  problems  to  all  participants.  The  following  sections 
summarize  the  procedures  used  in  each  test  session. 

Preliminary  Instructions  and  Procedures 

The  participants  were  tested  individually  and  sat  at  a  workbench  that  contained  the 
necessary  tools,  equipment,  and  technical  information  delivery  systems.  Instructions  were 
read  aloud  to  the  participants  to  provide  them  with  a  general  orientation  to  the  testing 
and  data  collection  methods.  Procedures  for  maintaining  the  technician's  anonymity  and 
voluntary  participation  in  the  testing  were  described,  followed  by  a  set  of  questions 
regarding  each  participant's  background. 

Since  the  technical  information  needed  to  solve  all  four  problems  was  available  in 
both  delivery  methods,  each  participant  was  randomly  assigned  to  a  four-problem  set  that 
counterbalanced  order  of  presentation  and  delivery  method.  Each  participant  then 
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received  instructions  on  the  basic  operation  of  the  automated  technical  information 
delivery  system  and  was  given  an  opportunity  to  practice  on  it.  Upon  completion  of  the 
preliminary  instructions  and  procedures,  which  averaged  30  minutes,  participants  were 
given  a  short  rest  break. 

Problem  Administration 

Each  troubleshooting  problem  began  with  a  brief  description,  read  aloud,  of  the  major 
casualty  symptom  (or,  in  technician's  jargon,  "gripe")  associated  with  the  printed  circuit 
card  failure.  This  information  was  followed  by  the  identification  of  a  specific  "start 
point"  in  the  technical  manual  or  the  computer  data  base  from  which  the  participant  was 
to  initiate  troubleshooting  actions.  In  addition,  participants  were  informed  that  the  test 
administrator  would  only  provide  information  to  them  regarding:  (1)  expiration  of  the 
time  allowed  for  completing  each  problem,  (2)  whether  a  printed  circuit  card  or  module 
they  recommended  for  replacement  would  or  would  not  correct  the  failure,  or  (3)  actions 
needed  to  extricate  themselves  from  difficulties  encountered  when  branching  through  the 
computer  data  base.  Participants  were  given  a  short  rest  break  between  each  trouble¬ 
shooting  problem. 

Debriefing 

The  test  session,  which  lasted  an  average  of  4  hours  per  technician,  ended  with  a 
discussion  of  the  purpose  of  the  study  together  with  a  general  review  of  performance  on 
the  problems.  The  technicians  were  then  thanked  for  participating  and  asked  to  refrain 
from  discussing  details  of  the  study  with  others  who  had  not  yet  been  tested. 

Data  Collection  Forms 

Three  basic  forms  were  used  to  record  the  performance  and  attitudes  of  the 
participants. 

Performance  Observation  Forms.  These  forms  were  used  by  the  test  administrator  to 
obtain  both  general  and  highly  specific  performance  data.  A  "general-purpose"  form 
provided  spaces  for  problem  start  and  stop  times,  particular  options  used  in  the  computer 
data  base,  and  other  information  related  to  test  problems.  Additional  space  was  provided 
on  this  form  for  recording  observations  and  comments.  Two  supplemental  forms  were 
used  for  recording  the  particular  troubleshooting  path  followed  by  the  participant  if 
he/ she  succeeded  in  localizing  the  fault  to  the  affected  module  and  card. 

User  Questionnaire.  A  user  evaluation  questionnaire  was  administered  to  each 
participant.  It  had  three  sections:  (1)  rating  of  physical  features  of  the  computer;  (2) 
evaluation  of  information  presentation  by  the  computer,  and  (3)  comparison  of  the 
presentation  modes.  The  section  on  physical  features  contained  questions  on  the  GRiDs 
keys,  display,  graphics,  and  scrolling  features.  The  focus  of  the  section  on  information 
presentation  was  adequacy  and  ease  of  use.  In  the  last  section,  the  technician  compared 
the  computer  presentation  mode  to  the  paper  mode  (Appendix  A). 

Structured  Interview.  In  addition  to  the  ratings  obtained  from  the  user  questionnaire, 
open-ended  comments  and  reactions  from  the  participants  were  recorded  by  the  test 
administrator  on  a  structured  interview  form.  The  first  four  items  dealt  with  various 
features  of  the  computer  data  base  and  the  next  four  addressed  the  users'  likes  and 
dislikes  concerning  the  use  of  the  prototype  electronic  delivery  device  for  supporting 
corrective  maintenance.  Spaces  were  also  provided  on  this  form  for  recording  any 
additional  comments  by  the  participants  (Appendix  B). 
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Experimental  Hypotheses  and  Design 

The  following  hypotheses  served  as  the  framework  for  comparing  performance  using 
the  two  methods  of  presenting  technical  information. 

1.  Troubleshooting  will  take  less  time  using  the  electronic  presentation  system  than 
using  technical  manuals. 

2.  Fewer  tests  will  be  performed  using  the  electronic  presentation  system  than 
using  technical  manuals. 

3.  Fewer  unnecessary  replacements  (modules  and  circuit  cards)  will  be  made  using 
the  electronic  presentation  system  than  using  technical  manuals. 

4.  More  faults  will  be  isolated  successfully  using  the  electronic  presentation  system 
than  using  technical  manuals. 

5.  Inexperienced  technicians  using  the  electronic  presentation  system  will  trouble¬ 
shoot  as  well  as  experienced  technicians  using  technical  manuals. 

6.  When  using  paper -based  technical  manuals,  experienced  technicians  will  trouble¬ 
shoot  better  than  inexperienced  technicians. 

7.  Performance  differences  will  be  greater  between  the  two  presentation  methods 
for  inexperienced  than  for  experienced  technicians. 

The  independent  variables  of  technician  experience  level  and  technical  information 
presentation  method  were  arranged  in  a  2  X  2,  between-subjects  multiple  analysis  of 
variance  (MANOVA)  design.  The  dependent  measures  were; 

1.  Time  to  isolate  the  failure  to  the  printed  circuit  card  level. 

2.  Time  to  isolate  the  failure  to  the  faulted  component  level. 

3.  Overall  time  to  problem  completion. 

4.  Number  of  external  test  points  checked  (i.e.,  test  points  accessible  without 
removing  modules  or  printed  circuit  cards  from  the  unit). 

5.  Number  of  discrete  components  checked  on  the  printed  circuit  card  containing 
the  fault. 

6.  Total  number  of  fault  isolation  tests  performed. 

7.  Number  of  false  replacements  recommended  to  correct  the  failures  (i.e.,  a 
participant's  recommendation  for  a  replacement  of  an  RT-728A/APX-64(V)  module  or 
printed  circuit  card  that  was  functioning  correctly  and  would  not  correct  the  failure). 

8.  Number  of  printed  circuit  card  failures  successfully  isolated. 


RESULTS 


Demographic  Comparisons 

Preparatory  to  the  tests  of  the  experimental  hypotheses,  analyses  were  conducted  to 
determine  the  extent  to  which  the  two  groups  of  participants  differed  in  terms  of 
responses  to  items  contained  on  a  personal  background  form.  These  analyses  were 
performed  to  provide  quantitative  evidence  that  the  two  groups  did  differ  on  variables 
relevant  to  experience  and  also  to  determine  if  any  demographic  factors  warranted 
inclusion  as  covariates  in  subsequent  data  analyses.  The  two  groups  were  compared 
according  to:  (1)  number  of  enlisted  occupational  specialties  held,  (2)  months  in  service, 
(3)  current  enlisted  pay  grade,  (4)  years  of  civilian  education  completed,  (5)  prior  civilian 
schooling/training  in  electricity  or  electronics,  (6)  prior  civilian  work  experience  in 
electricity  or  electronics,  (7)  prior  training/experience  in  operating  computers,  (8)  type  of 
RT-728A/APX-64(V)  maintenance  training  received  (formal  versus  on-the-job),  and  (9) 
months  of  experience  in  the  maintenance  of  the  RT-728A/APX-64(V). 

The  two  groups  differed  significantly  on  three  variables  relevant  to  experience:  The 
experienced  technicians  had  a  higher  enlisted  pay  grade  (4.8  versus  3.5),  longer  time  in 
service  (67.4  versus  44.2  months),  and  had  more  months  of  RT-728A/APX-64(V)  main¬ 
tenance  experience  (33.4  versus  2.8  months)  than  the  inexperienced  group.  Because  no 
reliable  group  differences  were  obtained  for  the  remaining  variables,  inclusion  of  these  as 
covariates  in  subsequent  analyses  was  not  considered. 

Performance  Analyses 

It  should  be  noted  that  9  of  the  36  participants  failed  to  isolate  the  faulted 
component  on  one  or  both  troubleshooting  problems  administered  when  using  maintenance 
manuals.  For  that  reason,  data  points  were  not  available  from  these  nine  participants 
with  respect  to  the  faulted  component  time  and  test  measures.  In  each  case  the 
troubleshooting  session  was  terminated  due  to  time  limitations  imposed  by  the  testing 
schedule.  (A  1-hour  time  limit  was  allowed  for  isolating  the  casualty  to  the  affected 
printed  circuit  card.  If  this  criterion  was  met,  an  additional  15-minute  time  period  was 
allowed  to  isolate  to  the  component  level;  otherwise  the  session  was  terminated.)  The 
absence  of  data  points  for  these  two  measures  also  precluded  the  use  of  the  overall  time 
and  the  total  number  of  fault  isolation  tests  for  these  9  technicians.  Consequently,  a 
decision  was  made  to  analyze  the  performance  data  in  two  ways.  The  first  MANOVA 
included  four  of  the  eight  categories  of  outcome  measures  for  the  entire  sample  of  36 
technicians  tested.  The  second  MANOVA  included  seven  categories  of  outcome  measures 
obtained  from  the  sample  of  27  technicians  who  successfully  completed  all  troubleshoot¬ 
ing  problems.  (The  number  of  problems  solved  was  excluded  from  the  latter  analysis  since 
this  measure  had  no  variance.)  Results  from  each  of  these  MANOVAs  are  presented  in  the 
following  sections. 

Overall  Sample 

Contrary  to  predictions  concerning  the  experience  factor  and  presentation  method 
(hypotheses  5-7),  no  significant  interactions  were  found  in  the  results  of  the  first 
MANOVA  (h[  =  36).  In  addition,  the  main  effect  for  experience  was  also  nonsignificant. 
Only  the  main  effect  for  presentation  method  was  significant:  F(4,  65)  =  16.28,  £<  .01, 
supporting  hypotheses  1,  3,  and  4.  The  difference  in  the  number  of  tests  performed  under 
each  method  of  presentation  was  also  significant,  but  the  result  was  the  opposite  of  the 
stated  hypotheses,  that  is,  there  were  more  tests  using  the  electronic  presentation 


system.  Table  2  provides  a  detailed  breakdown  of  the  performance  differences  attributed 
to  the  presentation  method. 


Table  2 

Performance  Difference  Resulting  from  the  Use  of  Technical  Manuals 
Versus  the  Use  of  the  Electronic  Presentation  System 
for  the  Overall  Sample 
(N  =  36) 


Measure 

X 

Technical 

Manual 

X 

Electronic 

Presentation 

F(l,  68) 

E 

Time  to  faulted  card  (min.) 

56.5 

24.4 

35.72 

.01 

Test  points  checked 

3.6 

5.6 

12.90 

.01 

False  replacements 

1.2 

0.0 

25.96 

.01 

Problems  solved 

1.7 

2.0 

9.90 

.01 

Reduced  Sample 

Results  from  the  second  MANOVA  (n  =  27)  mirrored  the  first;  only  the  main  effect 
for  presentation  method  was  significant:”  F(4,  65)  =  8.77,  £<  .01.  Results  are  shown  in 
Table  3. 


Table  3 

Performance  Difference  Resulting  from  the  Use  of  Technical  Manuals 
Versus  the  Use  of  the  Electronic  Presentation  System 
for  the  Reduced  Sample 
(n  =  27) 


Measure 

X 

Technical 

Manual 

X 

Electronic 

Presentation 

F(l,  50) 

E 

Time  to  faulted  card  (min.) 

45.6 

22.4 

20.77 

.01 

Time  to  faulted  component  (min.) 

28.5 

22.6 

3.99 

.06 

Total  time  to  solution  (min.) 

74.1 

45.0 

20.56 

.01 

Test  points  checked 

3.5 

5.0 

13.31 

.01 

Components  checked 

8.8 

11.5 

3.86 

.06 

Total  tests  performed 

12.3 

16.5 

8.34 

.01 

False  replacements 

0.8 

0.0 

11.79 

.01 
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User  Evaluation  Questionnaire 


The  results  from  the  31  items  in  the  User  Evaluation  Questionnaire  are  shown  in 
Table  4.  The  overall  response  to  the  computer  was  "highly  satisfactory"  or  "outstanding." 
Although  four  physical  characteristics  were  assessed,  a  single  mean  (3.95)  is  shown  for 
items  1  through  16  to  reflect  the  similarity  in  evaluation  of  the  four  characteristics. 
Each  of  the  16  items  was  consistently  rated  "highly  satisfactory"  or  "outstanding."  A 
single  exception  was  the  evaluation  concerning  the  adequacy  of  the  screen  size,  which  was 
rated  as  "satisfactory"  =  3.2).  The  adequacy  of  the  technical  information  presented  on 
the  computer  and  of  access  to  that  information  was  rated  "highly  satisfactory"  (X  =  4.22). 
Use  of  the  computer  was  perceived  as  an  improvement  in  terms  of  efficiency  and 
effectiveness  over  the  RT-728A/APX-64(V)  manual  for  troubleshooting. 


Table  4 

Summary  of  User  Questionnaire 
Evaluation 


Feature 


Questionnaire 

Items 


Physical  characteristics 
Information  presentation 
Efficiency 
Effectiveness 


1  - 

16 

3.95' 

17  - 

24 

4.22' 

25  - 

27 

1.681 

28  - 

31 

4.381 

Scale  values:  1  =  unsatisfactory,  5  =  outstanding. 

^Scale  values:  1  =  significantly  less,  5  =  significantly  more, 
^he  low  mean  for  items  25  -  27  is  a  positive  response. 


No  differences  were  found  between  the  ratings  of  experienced  and  inexperienced 
technicians.  Differences  in  the  evaluation  of  the  computer  were  obtained,  however, 
between  technicians  on  the  basis  of  previous  civilian  work  experience  in  electricity  or 
electronics.  Specifically,  technicians  with  no  previous  work  experience  provided  a  higher 
overall  rating  of  the  automated  system  (5!  =  4.3)  than  did  technicians  who  had  prior 
civilian  work  experience  in  electricity  or  electronics  (X  =  3.8). 

Structured  Interview 


A  summary  of  the  results  of  the  36  technician  interviews  is  shown  in  Table  5.  No 
data  are  shown  for  interview  items  1,  2,  4,  8,  and  9,  because  few,  if  any,  responses  were 
made  to  these  items.  Responses  from  all  participants  are  combined  since  no  differences 
were  found  between  the  services  or  between  experience  levels. 


Table  5 


i 


Summary  of  Interview  Responses 


Item 

Response 

n 

Percent 

#3:  Level  of  detail 

a.  Ease  of  switching 

Yes 

31 

86 

b.  Two  levels  sufficient 

Yes 

34 

94 

c.  Preference 

LESS 

9 

25 

MORE 

14 

39 

#5:  Likes  about  GRiD 

Quick/easy  access  to  information 

15 

42 

Proceduralized:  easy  to  go  from 

one  point  to  another 

12 

33 

Deletes  excessive  narrative; 

direct  path  to  fault 

9 

25 

Reduces  troubleshooting  time 

8 

22 

Ease  in  switching  among  frames 

7 

19 

#6:  Dislikes  about  GRiD 

Schematic:  cannot  see  entire 

schematic  on  the  screen 

17 

47 

Mi  Mode  preference 

GRiD 

27 

75 

Paper 

1 

3 

Both 

7 

19 

Level  of  Detail  (Item  #3) 


Technicians  assigned  high  value  to  the  provision  by  the  computer  of  two  levels  of 
detail  and  to  the  ease  with  which  they  could  switch  back  and  forth  between  levels. 
Experienced  and  inexperienced  technicians  generally  showed  no  difference  in  their 
preference  for  more  or  less  detail,  even  though  less  detail  was  suggested  as  better  for 
simple,  proceduralized  tasks  and  for  faster  troubleshooting  and  more  detail  was  suggested 
for  finding  parts  and  for  providing  extra  information  to  the  inexperienced  technicians. 

Likes  and  Dislikes  of  the  Computer  (Items  #5  &  #6) 

The  six  comments  shown  in  Table  5  express  the  scope  of  the  like-dislike  comments 
about  the  GRiD  and  account  for  95  percent  of  all  comments  provided.  A  general  theme 
was  that  the  computer  was  more  efficient  and  effective  in  providing  information  needed 
to  troubleshoot  the  faulted  printed  circuit  cards.  The  primary  dislike  concerned  the 
presentation  of  schematics.  The  most  frequent  comment  provided  in  this  regard  was  that 
schematics  were  too  small  (i.e.,  precluding  a  view  of  the  schematic  diagram  in  its 
entirety). 

Mode  Preference  (Item  if 7) 


The  computer  was  preferred  by  75  percent  of  the  technicians  for  supporting  fault 
isolation.  Nineteen  percent  of  the  technicians  expressed  preference  for  both  methods  of 
delivering  technical  information.  The  rationale  offered  by  technicians  who  preferred  both 
delivery  systems  was  that  the  technical  manual  would  be  used  when  referring  to  circuit 
schematics;  otherwise  the  computer  would  be  used  to  support  them  during  performance  of 


corrective  maintenance  actions.  Three  percent  of  the  technicians  indicated  that 
information  regarding  functional  descriptions  and  theory  of  operation  was  more 
adequately  presented  in  the  RT-728A/AP X-64(V)  technical  manual. 


DISCUSSION 

The  electronic  presentation  method  improves  troubleshooting  performance.  Using 
the  electronic  presentation  method,  technicians  took  less  than  half  the  time  to  isolate 
the  faulted  card,  made  no  false  replacements,  and  solved  ail  the  problems.  These  findings 
were  consistent  with  hypotheses  1,  3,  and  4.  Concerning  the  second  hypothesis,  more 
rather  than  fewer  tests  were  performed  when  participants  used  the  computer. 

In  formulating  the  second  hypothesis,  we  assumed  that  the  more  effective  trouble¬ 
shooting  strategy  would  be  one  that  incorporated  fewer  tests  (all  tests  being  relatively 
equal),  and  only  the  logic  in  devising  the  sequence  of  tests  would  differ.  When  using  the 
paper-based  technical  manual,  the  technician  had  to  rely  on  deductive  reasoning  to 
determine  where  and  what  type  of  test  point  to  check.  Presumably,  the  technician 
pursued  a  troubleshooting  strategy  that  provided  the  most  information  related  to  a  given 
symptom.  With  the  electronic  presentation  format,  on  the  other  hand,  troubleshooting 
was  more  directive,  that  is,  once  a  test  was  selected  and  the  results  entered  into  the 
computer,  the  computer  automatically  branched  to  the  next  logical  test  (derived  from 
subject  matter  experts'  analyses  of  the  fault)  in  the  fault  isolation  procedure  related  to 
the  given  symptom.  Results  demonstrated  that  even  though  technicians  using  the 
electronic  presentation  method  made  more  tests,  the  time  to  correctly  identify  the 
faulted  card  was  significantly  less. 

The  failure  of  the  experience  level  variable  to  account  for  any  appreciable  variance 
in  the  outcome  measures  when  using  the  electronic  presentation  method  is  consistent  with 
previous  research  findings.  For  example,  Foley  and  Camm  (1972)  found  that  when 
inexperienced  technicians  were  given  detailed,  step-by-step  maintenance  and  trouble¬ 
shooting  procedures,  they  performed  as  well  and  with  fewer  errors  than  more  experienced 
technicians  performing  the  same  tasks  as  they  normally  would  (i.e.,  with  or  without  using 
conventional  maintenance  manuals).  More  recent  evidence  to  support  the  present  findings 
was  obtained  in  the  AFHRL  maintenance  demonstration  described  previously  (Clay,  1986). 
This  study  found  no  difference  in  the  time  required  for  experienced  and  inexperienced  Air 
Force  technicians  to  perform  an  RT-728A/APX-64(V)  calibration  task  or  to  isolate  a 
faulted  component  on  one  of  its  printed  circuit  cards  when  using  the  computer. 

It  is  somewhat  harder  to  explain  the  failure  of  the  experience  level  variable  to 
account  for  any  appreciable  variance  in  the  outcome  measures  when  participants  used  the 
RT-728A/APX-64(V)  maintenance  manual.  Two  possible  factors  may  have  produced  these 
findings.  First,  while  results  indicate  a  clear-cut  difference  between  groups  with  respect 
to  the  average  amount  of  RT-728A/APX-64(V)  experience,  no  controls  were  imposed  on 
the  recency  of  that  experience.  Thus,  for  example,  some  technicians  had  3  years  of 
relevant  maintenance  experience  but  none  within  3  years  prior  to  the  testing!  In  fact, 
many  of  the  experienced  technicians  claimed  to  be  "rusty"  when  it  came  to  working  on  the 
RT-728A/APX-64(V),  a  factor  that  may  have  put  them  on  par  with  the  less  experienced 
technicians. 

Another  possible  explanation  has  to  do  with  the  particular  controls  imposed  during 
the  test  sessions  to  ensure  standardization  in  the  troubleshooting  approaches  used  by 


participants.  For  example,  participants  were  told  the  specific  location  in  the  mainte¬ 
nance  manual  from  which  they  were  to  initiate  troubleshooting  actions  and  were  also 
instructed  to  inform  the  test  administrator  when  they  wanted  to  replace  suspected  faulty 
modules  or  printed  circuit  cards  with  known  good  ones.  Because  of  this  structure  imposed 
on  the  problems,  participants  may  have  been  precluded  from  using  their  own  individual 
approaches  to  isolating  the  failures,  thereby  diminishing  any  differences  in  performance 
that  might  otherwise  have  been  demonstrated  by  their  experience  levels. 

Results  from  both  the  user  questionnaires  and  structured  interviews  indicate  that  the 
concept  of  electronic  delivery  of  technical  information  is  an  acceptable  alternative  to  the 
paper-based  maintenance  manual.  All  the  technicians  reacted  favorably  towards  the 
electronic  presentation  of  technical  information.  Using  the  paper-based  manual,  techni¬ 
cians  had  to  search  for  some  of  the  fault  isolation  information.  With  the  directive  nature 
of  the  electronic  presentation,  however,  the  search  behavior  was  ameliorated.  Although 
resistance  to  change  was  not  specifically  addressed  in  this  study,  the  overall  positive 
response  was  indicative  of  how  such  an  innovation  could  be  introduced  into  the  field.  In 
fact,  no  technicians  expressed  any  concern  about  the  electronic  and  mechanical  reliability 
of  the  computer  compared  to  the  omni-reliable,  paper-based  technical  manual. 

The  only  major  criticism  of  the  computer  concerned  the  inadequacy  of  the  schematic 
presentation  on  the  computer  display  screen.  It  is  interesting  to  note,  however,  that  most 
technicians  did  not  refer  to  the  schematics  when  using  the  computer.  Had  the  technicians 
elected  to  use  these  schematics  during  the  test  sessions,  it  is  possible  that  they  may  have 
evaluated  them  more  positively,  as  was  reported  by  Clay  (1986). 


CONCLUSIONS 

In  summary,  the  results  demonstrate  that  computers  can  be  used  as  an  effective 
means  to  present  technical  information  in  an  electronic  format.  If  the  technical 
information  data  base  is  constructed  for  ease  of  access,  as  was  that  for  the  RT- 
728A/APX-64(V),  maintenance  performance  should  improve.  More  importantly,  tech¬ 
nicians  appear  willing  to  change  to  a  different  delivery  method  for  obtaining  maintenance 
information. 

Since  the  data  base  was  very  limited  in  the  present  effort,  future  evaluations  need  to 
address  extremely  large  technical  information  data  bases  that  offer  many  ways  of 
entering  and  branching  to  the  various  types  of  information  within  those  bases. 
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USER  EVALUATION  QUESTIONNAIRE 
Introduction 

The  electronic  information  delivery  device  you  used  for  two  of  your  troubleshooting 
tasks  is  one  example  of  how  technical  information  may  be  delivered  in  the  future.  Since 
you  and  other  technicians  may  eventually  be  using  such  a  device,  your  reactions  and 
comments  are  important. 

Evaluate  the  questionnaire  items  using  the  5-point  scale  appearing  to  the  right  of  the 
items.  Rate  each  item  by  placing  an  "X"  in  the  appropriate  column.  Respond  to  as  many 
of  the  questionnaire  items  as  possible  but  recognize  that  there  may  be  some  items  you 
cannot  evaluate  based  on  your  limited  experience  with  this  device.  In  those  cases,  place 
an  "X"  in  the  column  head  "Can't  Evaluate.” 


As  Physical  Features 


Scale  Values 


Items 

Unsatisfactory 

Marginal 

Satisfactory 

Highly 

Satisfactory 

Outstanding  : 

Can’t  Evaluate 

1. 

Location  of  keys. 

2. 

Spacing  of  keys. 

3. 

Ease  of  operating  keys. 

B 

Indication  (feedback)  that  keys  had  been  activated. 

5. 

Reliability  of  keys  (i.e.,  Did  GRiD  respond  appropri¬ 
ately  to  the  keys  you  pressed?). 

6. 

Response  time  after  keys. 

B 

Adequacy  of  screen  size  for  displaying  information. 

S. 

Spacing  of  information  on  the  display  screen  (i.e., 
lack  of  clutter/crowding). 

Reference  Code 


Items 


9.  Brightness  of  display. 


10.  Contrast  between  information  on  the  display  and 
background. 


1 1.  Clare  on  display. 


12.  Legibility  of  displayed  letters  and  words. 


13.  Format/arrangement  of  graphic  displays  (e.g.,  sche 
matic,  block  diagrams,  IPBs). 


14.  Resolution  and  clarity  of  graphic  displays. 


15.  Adequacy  of  detail  on  graphic  displays. 


16.  Adequacy  of  scrolling  feature. 


Can't  Evaluate 
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B:  Information  Presentation 


Scale  Values 


Items 
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17.  Adequacy  of  the  organization  and  arrangement  of 
maintenance  information 


18.  Adequacy  of  options  available  on  menus. 


19.  Ease  of  using  menus  to  obtain  different  types  of 
maintenance  information  (e.g.,  procedural  steps, 
schematics,  IPBs). 


20.  Ease  of  obtaining  specific  information  within  a  par¬ 
ticular  type  of  maintenance  information  (e.g.,  locat¬ 
ing  a  specific  part  in  the  IPB). 

21.  Ease  of  obtaining  more/less  detailed  information. 


22.  Adequacy  of  illustrations  used  to  supplement  text  or 
procedures. 


23.  Adequacy  of  information  for  supporting  maintenance 
tasks  (i.e.,  completeness,  accuracy,  relevancy). 


24.  Ease  of  using  menus  to  return  to  the  appropriate 
place  in  a  set  of  procedures  after  branching  else¬ 
where  to  obtain  additional  information  or  details. 


Can't  Evaluate 
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C:  Com  para  tiave  Assessment 
(Electronic  versus  Paper  Documentation) 

This  section  of  the  questionnaire  deals  with  the  efficiency  and  effectiveness  of  the 
GRiD  computer  as  compared  with  current  maintenance  documents  for  the  APX-64.  The 
words  listed  under  the  heading  "scale  values"  have  changed.  Please  review  this  scale 
carefully  before  rating  the  items. 

To  avoid  repetition  in  the  wording  of  the  items  contained  in  this  section,  begin  each 
with  the  phrase: 

Compared  with  using  current  maintenance  documents  for  the  APX-64,  how  would  you 

rate  the  computer  version  in  terms  of  . . . 


Scale  Values 


Items 

Significantly 

Less 

Slightly 

Less 

No 

Difference 

Slightly 

More 

Significantly 

More 

25.  The  time  and  effort  required  to  change  from  one  type 
of  maintenance  information  to  another  (e.g.,  moving 
from  procedures  to  the  parts  information). 

26.  The  time  and  effort  required  to  obtain  more  detailed 
information  for  a  particular  maintenance  action  or 
step. 

27.  The  overall  time  and  effort  required  to  perform  fault 
isolation. 

28.  The  extent  to  which  it  represents  improvement  in  the 
overall  organization  and  arrangement  of  maintenance 
information. 

29.  The  ecxtent  to  which  it  represents  improvement  in  the 
presentation  of  maintenance  information. 

30.  The  extent  to  which  it  represents  improvement  in  the 
overall  completeness,  accuracy,  and  applicability  of 
maintenance  information. 

31.  The  extent  to  which  it  represents  improvement  in 
supporting  maintenance  for  the  APX-64. 

ft 


r 


APPENDIX  B 

STRUCTURED  INTERVIEW 
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STRUCTURED  INTERVIEW 

Based  upon  the  degree  of  use  of  the  following  features — 

scrolling 

schematics 

switching  between  levels  of  detail 
—ask  the  following: 

1.  If  scrolling  was  used: 

a.  Was  it  useful? 

b.  Was  it  easy  to  use? 

c.  Comments:  _ _ 


2.  If  schematics  were  used: 

a.  Were  they  easy  to  use? 

b.  Comments:  _ 


3.  If  more  than  one  level  of  detail  was  used: 

a.  Was  it  easy  to  switch  between  levels? 

b.  Were  two  levels  of  detail  sufficient? 

c.  Which  level  did  you  prefer? 

d.  If  no  to  "b," 

how  many  levels  should  there  be? 

e.  Why? 

f.  Comments:  _ 


5? 

/  u 

JiLii 
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If  no  switching  between  levels  of  detail  was  done: 


Ask  the  following  questions: 

5.  What  did  you  like  about  using  the  GRiD  personal  computer  for  maintenance 
information? 


8.  What  would  you  do  to  improve  the  way  maintenance  information  is  presented  on 
the  GRiD  computer? 


9.  Additional  comments: 


